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[57] ABSTRACT 

A feedback apparatus for an operator to control an 
effector that is remote from the operator to interact 
with a remote' environment has a local input device to 
be manipulated by the operator. Sensors in the effector’s 
environment are capable of sensing the amplitude of 
forces arising between the effector and its environment, 
the direction of application of such forces, or both am- 
plitude and direction. A feedback signal corresponding 
to such a component of the force, is generated and 
transmitted to the environment of the operator. The 
signal is transduced into an auditory sensory substitu- 
tion signal to which the operator is sensitive. Sound 
production apparatus present the auditory signal to the 
operator. The full range of the force amplitude may be 
represented by a single, audio speaker. Auditory display 
elements may be stereo headphones or free standing 
audio speakers, numbering from one to many more than 
two. The location of the application of the force may 
also be specified by the location of audio speakers that 
generate signals corresponding to specific forces. Alter- 
natively, the location may be specified by the frequency 
of an audio signal, or by the apparent location of an 
audio signal, as simulated by a combination of signals 
originating at different locations. 

15 Claims, 5 Drawing Sheets 
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operator’s hand acts as an additional input to the input 
APPARATUS FOR PROVIDING SENSORY device, thereby having an effect on the remote effector. 

SUBSTITUnON OF FORCE FEEDBACK It is also often desirable to amplify the force feedback 


U.S. GOVERNMENT INTEREST 5 

The U.S. Government has certain rights in this inven- 
tion pursuant to Contract No. NGT-50145, awarded by 
NASA and contract/Grant no. 956892 awarded by 
California Institute of Technology Jet Propulsion Lab. 

The invention relates in general to the field of non- 10 
visual, non-reactive displays of information. More par- 
ticularly, the invention relates to the display of informa- 
tion, such as force feedback, using non reactive sensory 
substitution displays, such as vibrotactile and auditory 
displays. 15 

BACKGROUND 

For many operations, a human operator manipulates 
an input device in an environment local to the operator, 
and the effect of the manipulation takes place in a re- 20 
mote environment, by virtue of the actions of an effec- 
tor located in the remote environment. Such operations 
include the manipulation of effectors: in outer space by 
ground based operators; in hazardous environments, 
such as a nuclear reactor core, by operators a safe dis- 25 
tance away; attached to construction equipment, such 
as cranes by operators distant from the crane. To facili- 
tate the control of the effector, various forms of feed- 
back have been used. Visual feedback, typically using a 
video camera and visual display device is very helpful. 30 
However, in many situations, visual feedback is not 
available, such as when the remote environment is dark 
or murky, for instance below water. Visual feedback 
can be degraded due to the various complexities in- 
volved in video transmission. Further, video transmis- 35 
sion requires a rather large bandwidth and relatively 
expensive equipment. Also, due to the arrangement of 
the effector with respect to the remote environment, the 
view of the visual capture device, such as a camera, is 
often blocked by intervening elements. 40 

It is also common to provide a physically reactive 
force feedback. In such an arrangement, the effector is 
provided with an array of force sensors, which are 
designed and located to measure the forces applied to 
the effector by interaction with its environment. These 45 
forces are transduced and transmitted to the environ- 
ment of the local operator, where they are retransduced 
by motor and gear systems into forces that are applied 
to the input apparatus that the operator uses. Thus, the 
operator “feels” the forces that the effector feels, to 50 
some degree depending on the fidelity of the force sens- 
ing and reproduction scheme. 

Force information is useful for several reasons. It 
facilitates governing the forces applied by the effector, 
either to avoid excessive stresses or to produce a desired 55 
movement of an object, or to minimize energy expendi- 
ture. It also facilitates estimating mass, frictional resis- 
tance, or other properties of objects in the remote envi- 
ronment in order to predict their behavior. It also pro- 
vides information about contact with objects or with 60 
features of the environment, as in searching when visual 
information is unavailable. In some cases, it also guides 
the operator’s body (typically the hand) to an appropri- 
ate location. 

Use of reactive force feedback has drawbacks. The 65 
apparatus required to reproduce the force at the local 
environment of the operator may be costly, compli- 
cated and bulky. Further, the force fed back to the 


signal, to enable sensing modest or delicate forces. If the 
gain on the feedback signal is large, in order to provide 
needed sensitivity, a small force at a remote location 
would be presented as a large force to the operator. 
This would impose a large force on the operator and 
could make the input controller more difficult to move. 
Thus the task would be made more physically difficult, 
the operation would be slowed down, and the rate of 
operator fatigue would be increased. If gains are made 
excessively high, instability could arise. 

Some force feedback systems present the feedback 
with a time delay. Time delays on the order of several 
seconds are common in connection with communica- 
tion to outer space, even with the signal being transmit- 
ted at the speed of radio waves. It has generally been 
found that reactive feedback in the presence of time 
delays can not be used effectively because instabilities 
quickly arise. See generally, W. R. Ferrel, “Delayed 
Force Feedback,: Human Factors , October 1986, pp. 
449-455. 

It has been noted that if reactive force feedback is 
provided to the operator, not to the hand that is apply- 
ing the force, but to the other hand, the operator gains 
an advantage. See S. Weissenberger and T. B. Sheridan, 
“Dynamics of human operator control systems using 
tactile feedback,” A.S.M.E. Transactions (J. Basic Eng.), 
vol. 84, pp. 297-301 (1962). However, the fact that the 
feedback is provided to another limb, and, in fact, an 
oppositely disposed limb, leaves room for improve- 
ment. 

It has also been attempted to provide force feedback 
through the visual modality, such as by using a visual 
display that indicates force, either through numbers, or 
a pictorial representation. However, the visual modality 
is typically heavily burdened by other demands. It has 
been found that such methods typically overload the 
operator’s ability to process visually perceived informa- 
tion, thus leading to degraded or unacceptable perfor- 
mance. 

There are other situations where feedback of force is 
desirable, which are not obviously within the class of 
remote effector applications. For instance, in microsur- 
gery, or any micromanipulation, the operator manipu- 
lates an input device with a resultant motion by an 
effector, perhaps very close to the operator, yet con- 
nected by a linkage (mechanical, electrical, electrome- 
chanical, or other) that effectively distances the opera- 
tor from the resultant activity as if the operator were far 
away. Such microsurgery could be conducted over 
distances of either inches or miles, with the appropriate 
visual feedback tools. Another instance relates to the 
control of artificial limbs, for instance an artificial arm 
or hand. One type of artificial limb is referred to as 
myeoelectric. Such a device is connected to the receive 
impulses from the nerves of the muscles remaining in 
the stump of the limb. The patient still has the ability to 
move these muscles, thereby generating electric signals, 
which can be used as inputs to the prosthesis. It would 
also be helpful if the patient were aware of the force 
field in which the prosthesis fingers, hand, wrist, elbow, 
etc. operates. Force related cues could provide valuable 
information to the patient. However, reactive force 
feedback is not practical in such a situation because the 
user has no sensation in the artificial limb. 
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Previous work of the present inventor has indicated 
that a vibrotactile display can be used to indicate 
whether or not a remote effector has come into contact 
with an object in the remote environment. In that work, 
a sensor/transducer carried by an effector was con- 5 
nected to a computer, which generated a signal to a 
vibrating unit fixed to the hand of the user that guided 
the input device. The computer generated signal was 
not related in any way to the magnitude of the force 
experienced by the sensor, other than that the signal 10 
indicated a positive force if the force exceeded the mini- 
mal sensitivity threshold of the sensor. No information 
was provided to the user about the magnitude or the 
direction of the force. See N. J. M. Patrick “Design, 
Construction, and Testing of a Fingertip Tactile Dis- 15 
play for Interaction with Virtual and Remote Environ- 
ments,” submitted in partial fulfillment of the require- 
ments for the degree of Master of Science in Mechani- 
cal Engineering at the Massachusetts Institute of Tech- 
nology pp. 2 and 63-67 (1990), citing the present inven- 20 
tor’s unpublished research. The entire document is 
hereby incorporated fully herein by reference. The 
principal focus of the Patrick work was using a vi- 
brotactile display to track the location of the user’s 
fingers relative to desired locations, particularly in the 25 
context of a virtual reality glove manipulation. The 
signal generated by the display device was a vibration 
having an amplitude linearly related to the distance 
between the location of the fingers and the desired 
location. 30 
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means for presenting the sensory substitution signal to a 
sensory system of the operator that is receptive to the 
signal. The full range of the force amplitude may be 
represented by a single, unitary display means, such as a 
mechanical vibrator or an audio speaker. 

In connection with the foregoing embodiment, vi- 
brotactile display elements can be located on the opera- 
tor’s limbs, such as on the hand, fingers, arms, legs, feet, 
etc. Auditory display elements may be stereo head- 
phones or free standing audio speakers, numbering from 
one to many more than two. 

According to another preferred embodiment, the 
invention includes an apparatus to generate a feedback 
signal such as discussed above, that corresponds to the 
location, relative to the effector and its environment, of 
the force between the effector and its environment. The 
location may be specified by the location of a vibrotac- 
tile display on the operator’s body, or the location of 
audio speakers that generate signals corresponding to 
specific forces. Alternatively, the location may be speci- 
fied by the frequency of an audio signal, or by the ap- 
parent location of an audio signal, as simulated by a 
combination of signals originating at different locations. 

Another preferred embodiment of the invention is a 
system, incorporating any of the feedback apparatus as 
discussed above, further including an effector and an 
input device. Additionally, the system may include a 
visual feedback system, such as a video camera at the 
effector’s location and a playback device at the opera- 
tor’s location. 


OBJECTS 

An object of the invention is thus to provide a non- 
visual, non-reactive sensory substitution force feedback 
signal for use in connection with remote effector appli- 
cations, which does not overload the capabilities of the 
operator and which remains stable even in the presence 
of time delays. A further object is to provide the advan- 
tages of force feedback without the burdens of reactive 
force feedback, such as an expensive and heavy or com- 
plicated effector and input device and instability in the 
presence of time delays. Another object of the invention 
is to facilitate increasing the gain of a force feedback 
system, in connection with the sensing of very small 
forces, without unduly increasing the resistance of the 
input device to operator control. Yet another object of 
the invention is to provide force feedback that is rela- 
tively inexpensive and robust and requires a minimal 
bandwidth to transmit the signal from the remote loca- 
tion to the local environment. 

SUMMARY 

A preferred embodiment of the invention is a feed- 
back apparatus for an operator to control an effector 
that is remote from the operator to interact with a re- 
mote environment. The operator manipulates a local 
input device. The feedback apparatus includes sensors 
in the effector’s environment capable of sensing the 
amplitude of forces arising between the effector and its 
environment, the direction of application of such forces, 
or both amplitude and direction. Means are provided 
for generating a feedback signal corresponding to such 
a component of the force, transmitting the feedback 
signal to the environment of the operator and for trans- 
ducing the signal into a non-visual, non-reactive sensory 
substitution signal to which the operator is sensitive. 
For example, the sensory substitution signal may be 
auditory or vibrotactile. The invention also includes 


BRIEF DESCRIPTION OF THE FIGURES 

These and other features, aspects, and advantages of 
the present invention will become better understood 
with regard to the following description, appended 
claims and accompanying drawings, where: 

FIG. 1 is a schematic representation of an embodi- 
ment of the invention using a vibrotactile display. 

FIG. 2A is a schematic representation of a force 
sensing resistor that may be used with the invention in 
plan view. 

FIG. 2B is a side cross-sectional view of the force 
sensing resistor shown in FIG. 2 A. 

FIG. 3 is a schematic representation of a number of 
force sensing resistors arrayed on a portion of an effec- 
tor. 

FIG. 4 is a schematic representation of an electronic 
circuit that can be used to amplify the signal generated 
by a force sensing resistor. 

FIG. 5 is a schematic representation of an embodi- 
ment of the invention using an auditory display. 

FIG. 6 is a schematic representation of a number of 
force sensing resistors arrayed on another effector. 

FIG. 7 is a schematic representation of a number of 
vibrotactile display elements arrayed on an operator’s 
hand. 

FIG. 8 is a schematic representation showing the 
local environment of an operator and the distant envi- 
ronment of the effector, with a visual feedback mecha- 
nism and means for controlling the effector and sensors. 

DETAILED DESCRIPTION OF PREFERRED 
EMBODIMENTS OF THE INVENTION 

In a preferred embodiment, the invention is an appa- 
65 ratus for providing force feedback through a sensory 
substitution display, for instance a vibrotactile display 
with a change in the amplitude of the vibration repre- 
senting a change in the magnitude of the force. A repre- 
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sentative embodiment of the invention is shown sche- 
matically in FIG. 1. A number of force sensors 
IO61-IO 6 m are used, the number depending on the effec- 
tor, the environment and the specificity of the informa- 
tion required by the task at hand. Suitable force sensors 5 
are described below. For each force sensor 106, an 
analog to digital converter (A/D) 104i„ m is provided. 
For instance, the A/D convertors can be implemented 
by a Data Translation DT-2801 input-output (I/O) 
board with sixteen analog to digital (A/D) channels and 10 
two digital to analog (D/A) channels. The board is 
installed in a programmable general purpose digital 
computer processor, such as an AT&T 6300 Personal 
Computer, which is used to process and thus augment 
the digital signals generated by the A/D converters 104. 15 
A number of digital to analog converters 108 \. n are 
connected to the output of the processor 102. Some of 
the D/A converters 108 may be implemented by an 
output board such as a Data Translation DT-2816 out- 
put board. The number of A/D channels used for a 2 
given application correspond to the number of force 
sensors needed for the task. The number of D/A chan- 
nels used depends on the number of display elements 
110. There need not be the same number of D/A chan- ^ 
nels as A/D channels, or as individual sensor signals. 

The D/A channels 108 on the DT-2801 and DT-2816 
output boards stimulate a number of vibrators 110 \. x 
that together constitute the vibrotactile display 112 and 
present the vibrotactile stimuli to the operator. The 
vibrators are attached to selected parts of the operator’s 
body, as described below. 

Suitable force sensors are force sensing resistors 
(“fsr’s”), as shown schematically in FIGS. 2A and 2B, 
such as made by Interlink Electronics of Santa Barbara 35 
Calif. As described in the Product Design Guide, avail- 
able from Interlink Electronics, the fsr 106 is a polymer 
thick film device which exhibits a decreasing resistance 
with increasing force applied normal to the device sur- 
face. A basic fsr has two polymer sheets laminated to- 4 0 
gether: one sheet 206 is coated with interdigitating elec- 
trodes 202 and 204. The other sheet 208 is coated with 
a proprietary semiconductive material. When a force is 
applied to the fsr 106, the semiconductive material 
shunts the interdigitating electrodes, causing the resis- 45 
tance to drop. Such resistors are described more fully at 
“Touch the Future Through Force Sensing Resistors 
from Interlink Electronics,” also available from Inter- 
link Electronics, which is incorporated fully herein by 
reference. Fsr’s are available in a variety of shapes and 50 
sizes. A typical range is an area of from 0.5 X 0.5 cm 
(0.2 X 0.2 in.) to 56x80 cm. (22x32 in.). Virtually any 
shape desireable is available, including squares, circles, 
etc. The sensitivity of such sensors ranges from ten 
grams to twenty kilograms. 55 

The fsr’s described above are useful for sensing forces 
that are normal to the surface to which the sensor is 
attached. For instance, as shown in FIG. 3, if such a 
sensor 306 1 were applied to the tip of a fingerlike effec- 
tor 302, it would be sensitive to forces experienced by 60 
the effector when pushing parallel to its long axis 
against an object 308, analogous to a human finger push- 
ing a button. It is also possible to use other types of 
sensors 3062, such as strain gauges, mounted along the 
side of such a finger-like effector, to sense and thus 65 
transduce bending stresses in the effector arising due to 
forces applied normal to the long axis of the effector, 
for instance parallel to arrows B\ and Bi. 


6 

It is necessary to provide an electric circuit that trans- 
duces the voltage output from the fsr’s 106 into a use- 
fully calibrated voltage spectrum. For example, a typi- 
cal effector may be able to apply forces to the environ- 
ment ranging from zero to ten lbs-f. With a current 
through the fsr limited to approximately 1 to 2 mA, an 
appropriate circuit is shown in FIG. 4. 

The power supply provides +12 volts and ground 
terminals. (A— 12 V terminal may also be provided.) 
The resistors 414 and 412, a trim potentiometer serve as 
a + 10 volt power source to the positive input terminal 
of the op-amp 410. To accomplish this the voltage into 
the op-amp is measured from ground up to the trim- 
potentiometer , and the trim-potentiometer is adjusted 
until the voltage is exactly 10 volts. 

The properties of op-amps dictate that the voltage at 
the negative terminal is + 10 V, since the voltage at the 
positive terminal is set to be + 10 V. The voltage across 
the fsr 406 is then 2 V. The resistance of each fsr may 
range, for example, from over 1 M-ohm at zero force to 
a smaller resistance at maximum force. For a typical fsr 
the minimum resistance is approximately 1500 ohms at 
five pounds of force. Thus the current through the fsr 
can range from approximately 0 A at zero force to 
approximately 1.3 mA at high force, which is within the 
suggested range for fsr current. 

For such an fsr 406, the circuit having a 4.7 k-ohm 
resistor 402 in series with a 0-5 k-ohm trim potentiome- 
ter 408 as shown in the upper part of the circuit dia- 
gram, allows voltage swings at V ow rfrom 10 V when no 
force is present to 0 V when a five pound force is pres- 
ent. These resistance and trim potentiometer magni- 
tudes are dependent on the characteristics of each indi- 
vidual fsr 406. The voltage equation for the upper por- 
tion of the circuit is 


\2V^V fsr +V X +V 2 +V 0Ul (1) 

where V f sr is the voltage across the fsr, which equals 2 
V; Vi is the voltage across resistor 402, V2is the voltage 
across the in loop (left) portion of the potentiometer 
408; and V out is the op-amp output voltage, which cor- 
responds to the voltage across the portion of the trim 
potentiometer 408 not connected in series with the resis- 
tor 402. The above equation (1) then reduces to 


Vi + V 2 +V out =10V. (2) 

Since an op-amp draws no current, the current through 
the resistors 406, 402 and 408 equals 


(Vfsr/Rfsr)=(2V/Rf sr ), (3) 

where R/ Sr is the resistance of the fsr 406 measured at 
any particular time. The equation (3) reduces to 

V out = 1 0V — (4.7 k-ohm)(2 V/R fsr )-(R 2 ) (2 V/R fsr ). (4) 

When there is no force at the fsr 406, R/ 5r is very large 
and V 0 ut equals ten volts. The correct value to which 
the trim potentiometer 408 is adjusted depends on the 
characteristics of the particular fsr 406. The fsr resis- 
tance value at maximum force is substituted for R/ sr and 
W out is set to be zero. The trim potentiometer 408 is 
tuned to insure that V 0 ut is zero volts at maximum force, 
giving a swing of 10 V to zero V for changes in force 
from zero pounds to maximum force. For example, if 
R/sr were 1400 k-ohm at maximum force, the trim poten- 
tiometer 408 resistance in series with the 4.7 k-ohm 
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resistor 402. would be set to approximately 2.3 k-ohm, 
giving a Y 0 ut of zero at maximum force. 

For a typical display, the number of fsr’s used may 
range from two to nine, however, even more sensors 
can be used. A suitable vibro tactile display electronics 
unit is described in detail by Patrick, cited above. 
Briefly, the vibrotactile display, consists of vibrating 
voice coils. Suitable vibrators are available from Audio- 
logical Engineering Corporation in Somerville, Mass. 
The displays vibrate at 250 Hz, which is the frequency 
at which the Pacinian corpuscles (the rapidly adapting 
vibration detectors in the skin) have their lowest thresh- 
old. The resonant frequency of the vibrators is also 250 
Hz. Therefore the highest efficiency of voltage into the 
display and vibration amplitude is achieved. Suitable 
circuitry is provided as part of vibrotactile display 112 
to convert the voltage output from D/A convertors 108 
into a periodic signal suitable for excitation of vibrating 
elements 110. Design of such a circuit is within the 
competence of one of ordinary skill in the art and is 
described fully in Patrick, cited above. 

As shown schematically in FIG. 7, the vibrotactile 
displays may be placed on various body locations, such 
as on the fingertips 710 1 and 7102, palm of the hand 7103, 
and wrist 71(4, depending on the number and location 
of critical forces present for an experimental task. A 
simple joystick type input device 720 is shown in FIG. 
7, however, more sophisticated input devices can be 
used, such as a joystick in combination with a pair of 
spring loaded arms or a virtual reality type glove, such 
as are available from VPL Research of California, 
under the tradename “Dataglove.” 

Placement of the vibrating elements on the fingertips 
and palms is for two reasons: 1) they are the regions of 
the skin which are most sensitive to vibrotactile stimula- 
tion (see Wilska, A. (1954). “On the Vibrational Sensi- 
tivity in Different Regions of the Body Surface,” Acta 
Physiologica Scandinavica, Vol. 31, pp. 285-289) and 2) 
they allow the displays to be placed in positions which 
allowed easy transformations through sensory substitu- 
tion for the operator to identify the position of the 
forces exerted at the remote effector location. The vi- 
brators may be held in place on the skin using nylon 
hook and loop fasteners. 

According to a preferred embodiment, the magnitude 
of vibration depends on the magnitude of the force 
being displayed by the vibrating element. Research 
studies have determined relationships for the perceived 
magnitude of a vibrotactile stimulus as a function of the 
physical intensity of the stimulus. The physical intensity 
is the displacement of the vibrator which is usually 
measured in microns. A study by Verrillo, R. T. and 
Chamberlain, S. C. , “The effect of Neural Density and 
Contactor Surround on Vibrotactile Sensation Magni- 
tude.,” Perception and Psychophysics , Vol. 11 (IB), pp. 
117-120 (1972) (which is incorporated fully herein by 
reference) shows a relationship between displacement 
and sensation magnitude according to the function: 


(SM) 0A *=k*D, (4) 

where SM is sensation magnitude, and D is displace- 
ment in db and k is a constant determined by measure- 
ment. Db’s are related to displacement in microns by: 

db=2O/0g(f% (5) 

where P is the displacement in microns. The relation- 
ship between voltage in and displacement P can be 
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calibrated. Converting displacement in db to displace- 
ment in microns, applying the V/ B to displacement func- 
tion, and linearly scaling subjective magnitude between 
0 and 5 pounds for x and z-directions and between 0 an 
5 10 pounds for the y-direction, gives the following rela- 
tionships between force and input voltage: 

F/ n =0.0493*10 a748 V^£0.48_ 0049 3 (6) 

10 for forces in x and z-directions and 

F/„=0.0493*10 a536 V^.48_ 0i049 3 (7) 

for forces in the y-direction. The (—0.0493) is included 
15 so that zero force would correspond to zero volts. The 
relationship allow a change in force to be perceived as 
a change in the magnitude of the vibrotactile stimulus 
and are used in software routines that control the pro- 
cessing by the processor 102. The software is relatively 
20 straightforward and is well within the scope of the skill 
of one familiar with the art. 

The specific conformation of the implementation of 
the invention depends on the task to be accomplished, 
which in turn governs the shape, degrees of freedom 
25 and capabilities of the effector as well as the control 
mechanism. For instance, a typical effector is shown 
schematically in FIG. 6. The effector has many degrees 
of freedom (“dof ’) plus a gripping capability. Three of 
the degrees of freedom, designated X, Y and Z, identify 
30 arm translation. One (“Azimuth”) identifies arm rota- 
tion. Another (“Elevate”) identifies gripper rotary ele- 
vation and another (“Twist”) specifies gripper twist. 
Jaw grasping is indicated at “Jaw.” The arm can also 
rotate about the X, Y and Z axes. 

35 Force sensing resistors may be placed at various loca- 
tions on the effector. For instance, as shown schemati- 
cally in FIG. 6, sensors 606 1 and 6O62 are located on the 
inside, or grasping surfaces of the jaw members. Such 
sensors would provide information about the force of 
grasping, such as for holding delicate objects. Sensors 
6O65 and 6066 are located on the outside surfaces of the 
jaw members. Such sensors would provide information 
concerning the jaw contacting the environment, or the 
45 amount of spreading forces that the jaw members may 
be providing. Sensors 6O63 and 6O64 are located on the 
end surfaces of the jaw members. These sensors would 
provide information concerning the force with which 
the gripper may be pushing against an object in the X 
50 direction directly ahead of it. Sensors 6O67, 606s and 
6O69 are located on the arm of the effector. These sen- 
sors may be a different sort of sensor, such as a strain 
gauge, that measure bending strains in the arm. They 
may provide information concerning the downward 
55 force being applied to the arm, such as pertaining to the 
weight of an object being grasped by the jaws. 

Signals from the individual sensors may combine to 
provide information pertaining to the direction of an 
applied force. For instance, if both sensors 6O63 and 
60 6O65 indicate a force of certain respective magnitudes, 
an operator might conclude that the gripper member 
was pushing against a comer, or portion of the environ- 
ment having two surfaces at an angle to each other. The 
amount of information that the sensors can provide 
65 depends to some extent on the familiarity the operator 
has with the environment in which the effector oper- 
ates. For instance, if an effector is operating in an envi- 
ronment of known configuration, but having some por- 
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tions for which available visual feedback is obstructed, 
much information can be gained from knowing which 
portions of the effector are contacting the environment. 
Such situations might include locations associated with 
a nuclear reactor, or certain medical situations where 
other imaging has already provided a “geography” of 
the locus of interest. 

The manner in which signals from the sensors are 
presented to the operator also depends on the situation. 
The system should be designed so that the operator can 
associate the stimulus signals with the environment they 
simulate. For instance, as shown in FIG. 7, vibrotactile 
display element 710i, located on the inside of the opera- 
tor’s index finger, might be driven to represent forces 
sensed by fsr 6O61 as shown in FIG. 6, on the inside of 
the gripper member. Similarly, display element 7102, 
located on the fleshy surface of the operator’s thumb, 
might be driven to represent forces sensed by fsr 6O62, 
located on the inside of the opposite gripper member. 
Sensor element 7103, located on the operator’s palm, 
might be driven to represent forces sensed by fsr 6066, 
located on the outside of one gripper member, with 
another display element, not shown, on the back of the 
user’s hand, driven to represent the force experienced 
by fsr 6O65, on the outside of the other gripper member. 
Finally, display element 7104, located on the user’s 
wrist, may be driven to represent the force experienced 
by one or a combination of strain gauges 6O67, 606s and 
6O69, thereby providing information about the weight of 
an object grasped by the effector. 

The choice of association between locations of fsr 
sensors on the effector and display elements used by the 
operator is virtually limitless. The designer should con- 
sider the sensitivity of the portion of the body contact- 
ing the display. The designer should also consider simi- 
larities in the conformation of the effector and the body 
member, so that it is within the user’s capability to make 
the association between the display and the stimulation. 
The number and complexity of display elements must 
not be so great as to overload the user’s ability to pro- 
cess the information presented. 

As is mentioned above, it is helpful to vary the vibra- 
tional intensity of the display elements relative to the 
force sensed in such away that the user perceives a 
doubling of the sensation when a doubling of the force 
arises. Thus, a linear relation between force and vibra- 
tional amplitude is usually not warranted, as discussed 
in Verrillo and Chamberlain [1972], cited above. Rather 
a relation that is linear between sensation magnitude 
and force is preferred. 

The foregoing has discussed using a vibrotactile sen- 
sory modality as a substitution for force feedback in 
connection with a master slave operator. It is also possi- 
ble to use the auditory modality as a substitution for 
force feedback. Such use is shown schematically with 
respect to FIG. 5. The system is the same as the system 
shown in FIG. 1, with respect to the tactile modality, 
except that the display is auditory rather than vibrotac- 
tile. Like elements in both systems are identified with 
like reference numerals. The common elements include 
the array of force sensing resistors 106, the analog to 
digital channels 104 and the programmable digital com- 
puter 102 equipped with input and output boards. An 
additional element is a C/MS PC music audio card 540, 
available from Creative Labs, Inc., of San Jose, Calif. 
The card can be used to independently control up to 
twelve different audio signals. Each signal has a single 
square wave frequency. The card has 4 bits for ampli- 
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tude control of a nominally right channel and 4 bits for 
amplitude control of a nominally left channel for each 
of the 12 signals. This enables stereo presentation. There 
are also eight bits for frequency control of each of the 
12 signals, and 3 bits for octave control. The frequency 
and octave ranges produce a frequency range from 20 
to 7000 JHz. This range contains the frequencies at 
which the human ear is most sensitive and the funda- 
mental frequencies that correspond to all of the notes on 
a piano keyboard. The capabilities of the board allow a 
change in force at any of the fsr’s 106 to be presented to 
the subject with frequency and apparent location and- 
/or frequency of the tone (discussed below) identifying 
the position of the force, and with amplitude (loudness) 
of the tone representing the magnitude of the force. 

Intensity is a physical property of sound (the rate of 
energy transfer), whereas loudness is a subjective psy- 
chophysical property. Loudness is the sensation magni- 
tude or perceived intensity of a sound, and it increases 
with sound intensity (see generally BofF, K. R., and 
Lincoln, J. E., Engineering Data Compendium: Human 
Perception and Performance . AAMRL, Wright-Patter- 
son AFB, OH (1988)). Changes in loudness make per- 
ceptual differences to humans, and the design of the 
auditory loudness display includes the determination of 
a relationship to represent a change in force as a change 
in loudness, rather than a change in sound intensity. 
This is directly analogous to the care taken to insure a 
linear relation between tactile sensor magnitude and 
force. 

Several experimental studies have been conducted to 
determine how loudness depends on stimulus intensity 
or level. Most of these studies were conducted with a 
1,000 Hz tone, since this is the frequency at which the 
ears have the lowest threshold when earphones are 
worn (see generally Scharf, B. and Buus, Soren, “Audi- 
tion I: Stimulus, Physiology, Thresholds,” Handbook of 
Perception and Human Performance , Chapter 14. Eds. K. 
R. Boff, L. Kaufman, and J. P. Thomas. New York: 
John Wiley and Sons (1986)). For these reasons, a 1000 
Hz tone is selected for the invention to represent forces 
whenever practical. However, for any specific operator 
and operator environment, other frequencies may be 
more useful, as one of ordinary skill in the art will 
readily appreciate. The relationship between loudness 
and sound pressure level in decibels (Scharf, B. and 
Houtsma, A. J. M., “Audition II: Loudness, Pitch, Lo- 
calization, Aural Distortion, Pathology,” Handbook of 
Perception and Human Performance , Chapter 15. Eds. K. 
R. Boff, L. Kaufman, and J. P. Thomas, New York: 
John Wiley and Sons (1986)) follows the relationship: 

1 ^ 0 . 0474 * 10°-° 315 5 ( 8 ) 

where 1 represents loudness in sones and s represents 
sound pressure level in dbs, with R equal to 1.00. The 
value R is a measure of the closeness of the fit between 
a curve and the data it represents. A value for R of 1.00 
indicates the closest fit. The auditory signal is presented 
to the operator through speakers 520 1, 5202, • • • 520*, 
with x running from zero. It is convenient to configure 
the speakers 520 as a pair of headphones, however, 
other arrangements with more speakers are possible. 

For instance, the operator can be surrounded with a 
number of speakers, all in the same general plane, or in 
several planes, with some speakers above and some 
below the operator. The limit on the number and place- 
ment of the speakers must be determined by cost and 
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computational limitations, the complexity and impor- 
tance of the task to be performed, as well as the opera- 
tor’s capacity to process the variety of acoustic signals 
produced by the speakers. If more than two speakers 
are used, it is also important to locate the speakers in a 5 
way that corresponds to the arrangement of force sen- 
sors, the effector, and the effector environment. In the 
discussion that follows, the two speaker example of a 
pair of headphones is used, however, the generality of 
the invention is not so limited, and those of ordinary 10 
skill in the art will readily understand the means by 
which the apparatus described can be modified to ac- 
commodate numerous speakers. 

Suitable headphone speakers are available as small 
earplugs from Radio Shack, which fit snugly into an 15 
operator’s ears, having a frequency response range of 
sixteen Hz to twenty-two thousand Hz. Ear muffs, such 
as sold under the tradename Viking by the Bilsom 
Safety Products Company of Sterling, Va., may be 
worn over the earplugs to cut down on environmental 2( 3 
noise and to increase the efficiency of the auditory stim- 
ulus. 

The C/MS card 540 has four bits for loudness control 
through a loudness control register. Therefore, the 
input value to the loudness control register may range 25 
from zero to fifteen in decimal units, where zero corre- 
sponds to zero loudness and fifteen corresponds to max- 
imum loudness. The auditory display is calibrated with 
the C/MS card, and the relationship between the input 
value for the loudness control register in decimal units 30 
for the C/MS card and the corresponding db sound 
pressure level (“SPL”) output from the ear plugs is 
according to the following: 

a— 1.97*c 0 - 6 (9) 35 

where a is auditory intensity (db SPL), c is input value 
to the card loudness control register, in decimals and 
R=1.00. Sound pressure level db in decibels may be 
related to sound pressure P in micropascals by the fol- 40 
lowing: 

db= 20* log (P/20). (10) 

Converting sound pressure level from equation (9) in db 45 
to sound pressure level in micropascals provides: 

£=*V 6 ) (11) 

where L is loudness in sones, k=0.01, and P is the sound 50 
pressure in micropascals. The sone is arbitrarily scaled 
and is defined as a unit of loudness such that a 1,000 Hz 
tone heard binaurally at 40 db has a loudness of one 
sone, according to the method of Scharf and Houtsma, 
1986 cited above. 55 

The C/MS board 540 has four bits for volume con- 
trol, i.e., sixteen different levels for the volume register 
value. Further, sound pressure from the board is lin- 
early proportional to the volume register value. There- 
fore volume register value is substituted for sound pres- 60 
sure, P, in the above mentioned equation no. (11) and 
allowed to vary between zero and fifteen. Further, 
loudness in sones is linearly scaled to force between 
zero and five pounds for translation forces in two di- 
mensions of a plane, designated x and z, and from zero 65 
to ten pounds for translation forces perpendicular to the 
plane, designated in the y direction. This produces an 
equation: 


F=k*(VR^- 6 ) (12) 

where F is the force in pounds, VRV is the volume 
register value in decimal numbers, and k is a constant to 
be determined and dependent on the force range. By 
letting zero force be represented by zero volume regis- 
ter value and five or ten pounds force (depending on the 
degree of freedom) be represented by a volume register 
value of 15, k is estimated at 0.95 for 0 to 5 pounds and 
at 1.97 for 0 to 10 pounds. These functions relate 
changes in the volume register value of the C/MS 
board, i.e., a change in sound pressure, to a change in 
the perceived force by varying the perceived loudness 
of the tone. It is then possible to develop the relation- 
ship that sets the loudness level of the tone based on the 
force reading on an fsr 106. This relationship is as fol- 
lows, for the x and z direction forces: 

VR V(decimal)= 1.026 */orce L61 (13) 

where R=1.00. The relationship for the y direction 
forces is: 

VKV (decimal) =0.322 *force 1 - 67 * (14) 

If equations (13) and (14) were used as presented, the 
volume register values would be zero (producing no 
sound at all) until the force equals 0.985 pounds in the x 
or z directions and 1.97 pounds in the y direction. The 
minimum force registrable by a typical fsr 106, how- 
ever, is 0.01 pounds (for the fsr’s identified above). 
Thus, in order to have the display provide stimulation at 
low force levels, the equations (13) and (14) can be 
shifted to provide an auditory stimulus at lowest loud- 
ness for the minimum force measurable. The following 
equations may be used: 

VR V(decimal)= 1 .026* ((force + 0. 97972) 1 - 67 ) (15) 

and 

VR V(decimal)=0.323*(force+l.96503) L61 ). (1 6) 

At low force levels, the display may hiss, fluttering 
between on and off. To overcome this problem, a digital 
filter may be used to average a number of previous 
readings, for instance four, from an fsr 106, to determine 
the force value. Since the sampling rate is 60 Hz, this 
does not significantly slow down the effective sampling 
rate. 

The overall set up for a generic sensory substitution 
display for force feedback is shown schematically in 
FIG. 8. An effector 840 is equipped with fsr’s 806. The 
effector operates within an environment 860. The effec- 
tor is connected by some appropriate channel, indicated 
by dotted lines 842, to a control device 802, such as a 
properly programmed digital computer. The fsr’s 806 
are also connected through a channel, indicated by 
dotted lines 844, to a control device, which may conve- 
niently be the same control device 802 as controls the 
effector 840, or it may be another control device more 
specially tailored to the control of the fsr’s. 

The channel may be a plurality of electrical wires, or 
fiber optic cables. Alternatively, the channel may be a 
radio frequency (or light frequency, i.e. laser) channel, 
for instance as may be used to communicate with an 
effector located on a satellite in outer space, controlled 
by an operator on earth. 



5,451,924 


13 

Visual feedback is provided by image capture device 
852, for instance a video camera, connected through an 
appropriate channel to an image display device 854, 
such as a cathode ray tube, located in the local environ- 
ment of the operator. 5 

At the local environment of the operator, the sensory 
substitution display 812 provides force feedback 
through a modality other than reactive force, such as 
vibrotactile or auditory. The display elements 870 are 
applied to the operator in the appropriate fashion, de- 10 
pending on their modality. They are under the control 
of a control device 802 , again conveniently embodied in 
a digital computer. The operator provides manual input 
signals to an input device 820 , such as a levered joystick 
or an actuator glove, which generates signals that are 15 
used by the control device that controls the effector 
840 , such as digital computer 802 . 

Use of sensory substitution displays has proved bene- 
ficial in certain circumstances. Evaluation of the success 
of such use may be considered in light of certain crite- 20 
ria. 

For instance, one measure is the number of absolute 
judgments that the user can make based on the substi- 
tuted stimuli. Four test subjects each began by trying to 
distinguish among three different frequencies (or, in a 25 
separate test, amplitudes). As the session continued, the 
number of stimuli in a grouping were increased to four, 
five, six, seven, and ten different stimuli. Each subject 
received each stimulus within a group four times for a 
total of sixteen x n trials per group, where n was the 30 
number of stimuli within a group. The subjects were 
trained and given practice in recognizing different fre- 
quencies (or amplitudes) and assigning numbers to each 
stimulus in a group. Each stimulus was presented for 2.5 
seconds. The subject then responded by assigning a 35 
number to that stimulus. After responding, the subject 
was told whether or not the response was correct, and 
if it was incorrect the subject was told the correct re- 
sponse. This process continued until all trials for that 
group was completed. Before each group of tones was 40 
presented, the subject was given some warm-up and 
practice trials in order to more easily adjust to the in- 
creased number of stimuli in the new group. 

The inventor conducted channel capacity experi- 
ments using auditory stimuli varying in loudness to test 45 
the auditory display. The stimuli were spaced by equal 
sensation magnitude based on the sone scale (see above) 
between 46 to 70 db SPL. A 1000 Hz auditory tone was 
presented to both ears simultaneously. A channel capac- 
ity of approximately 1.76 bits representing approxi- 50 
mately 3.4 absolute judgments was determined. 

Auditory frequency may also be used to provide 
information about force feedback, particularly when a 
force can be applied to the effector in more than one 
direction. Test subjects were presented with a number 55 
of tones varying in frequency and were asked to assign 
a number to each tone. The number of tones (different 
pitches) were increased until the channel capacity was 
determined. The stimuli were spaced equally on a loga- 
rithmic scale from 28 to 3500 Hz. The tones were simul- 60 
taneously presented to both ears at an intensity of ap- 
proximately 65 db spl. The results show a channel ca- 
pacity of approximately 2.18 bits or 4.5 absolute judg- 
ments. 

Similar amplitude experiments have been conducted 65 
with the vibrotactile display discussed above. The vi- 
brotactile display allowed amplitude modulation of a 
250 Hz vibratory signal with an amplitude range of 7 to 
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32 db (i.e ... 1 micron). This range was divided into 
sections of equal sensation magnitude for the absolute 
judgments experiments. The 250 Hz vibratory signal 
was presented simultaneously to the index finger and 
thumb. A channel capacity of approximately 1.76 bits 
representing approximately 3.4 absolute judgments was 
determined. 

In many teleoperation tasks an operator needs to 
distinguish whether an applied force is increasing or 
decreasing over time. In order to calibrate the effective- 
ness of the vibrotactile and auditory displays to present 
this information, tests on relative discrimination to de- 
termine just noticeable differences were conducted. 
These tests determined the operator’s ability to detect a 
small difference between two stimuli or a change in one 
stimulus. The least change in a stimulus or the least 
difference between two stimuli that can be detected is 
called the Just Noticeable Difference (JND) which is 
also expressed as AI to represent a change in intensity. 

The C/MS card 540 had four bits for volume control, 
providing fifteen discrete loudness levels between 46 
and 70 db SPL. Therefore a maximum of 14 JND’s were 
possible. These levels were tested and all fifteen levels 
were easily identifiable when presented relatively from 
one to the next, thus providing 14 JND’s for the audi- 
tory loudness display. 

Similar experiments conducted with the vibrotactile 
display show approximately fifteen JND’s within the 
range of zero to forty microns. To some extent, the 
noticeability of a difference depends on the location 
along the amplitude spectrum. See generally Stevens, S. 

5., “Mathematics, Measurement, and Psychophysics,” 
S. S. Stevens (Ed.), Handbook of Experimental Psychol- 
ogy. New York: John Wiley & Sons (1951) and Rabino- 
witz, W. M., Lim, J. S., Braida, L. D., and Durlach, N. 

1., “Intensity Perception VI: Summary of Recent Data 
on Deviations from Weber’s Law for 1000 Hz Tone 
Pulses,” Journal of the Acoustical Society of America, 
Vol. 59, No. 6, June, 1976, pp. 1506-1509 (1976). 

Experiments conducted by the inventor have shown, 
in general, that for most applications, conventional 
reactive force feedback allows the operator to accom- 
plish tasks more quickly or more precisely than does the 
use of either auditory or vibrotactile sensory substitu- 
tion force feedback. However, as has been mentioned 
above, reactive force feedback is expensive, and adds to 
the bulk, weight and complexity of systems requiring its 
use. Thus, for instances where a tradeoff is acceptable 
between speed or accuracy on the one hand, and cost, 
bulk, weight, complexity, etc. on the other hand, sen- 
sory substitution according to the present invention 
provides distinct advantages. 

Vibrotactile and auditory sensory substitution force 
feedback show a distinct advantage over conventional 
reactive force feedback in the presence of a time delay, 
such as would be present in connection with control of 
an effector in outer space. As is well known, and men- 
tioned above, use of reactive force feedback is impossi- 
ble in the presence of a time delay because instabilities 
arise. The instabilities arise because the reactive feed- 
back is applied to the operator’s body, typically the 
hand, which is manipulating the input device. Thus the 
time delayed force feedback itself becomes an input to 
the system, quickly resulting in instabilities. Neither the 
auditory nor the vibrotactile force feedback signals 
result in a direct input signal to the operator’s input 
device. Thus, the instabilities do not arise. 
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The vibrotactile and auditory modalities of force 
feedback are particularly useful in those situations 
where visual feedback, which is common in such sys- 
tems, is unavailable, or impaired or otherwise degraded. 
Such situations would include underwater operations, 5 
or operations in dirty atmospheres, or operations where 
physical portions of the environment simply block the 
view of the image capture device from time to time. 
Moreover, high fidelity video feedback requires a sig- 
nificant signal bandwidth. Use of the sensory substitu- 10 
tion according to the invention permits use of lower 
fidelity video feedback, requiring a smaller bandwidth. 

Another reason that vibrotactile and auditory sensory 
substitution force feedback have an advantage over 
reactive force feedback is that lower force thresholds 15 
are detectable using these substituting modalities. This 
is because very small force signals can be amplified 
using the control systems, without affecting the opera- 
tor’s ability to provide an input signal to the system. 
With a reactive feedback system, if very small forces are 20 
amplified so that the feedback signal is noticeable to the 
operator, this necessarily means that larger forces will 
also be amplified, thus, making it more difficult for the 
operator to move the input device in the presence of 
these large forces. Thus, for simple tasks where all that 25 
is required is to know whether or not contact between 
the effector and the environment has been made, vi- 
brotactile or auditory sensory substitution force feed- 
back may provide operational advantages over reactive 
force feedback, even in the absence of a time delay and 30 
with unimpaired visual feedback. 

An additional advantage of the sensory substitution 
modalities of force feedback is that they do not invoke 
in the operator’s hand or arm (or other limb implement- 
ing control) a reaction. Thus, the operator can more 35 
easily control the input device, without needing to com- 
pensate for the reaction to the force feedback. 

There are many applications in which increased sensi- 
tivity of force detection could be useful. For example, if 
a physician were performing a surgical procedure in 40 
which knowing the amplitudes of small forces being 
applied to the patient were helpful, sensory substitution 
could provide advantages. For a small force applied to 
the patient, an above threshold stimulus could be pres- 
ented through sensory substitution. If the same tech- 45 
nique were used with a high feedback gain for tradi- 
tional force feedback, movements could be slowed or 
made overly difficult. High sensitivity to small forces 
could also be useful for a telemanipulation task involv- 
ing a delicate instruments or one that requires the appli- 50 
cation of small forces for successful task completion. 

The mode of implementation of a sensory substitution 
display depends upon the environment, effector, input 
device and other criteria. Considering first an auditory 
modality, numerous fsr’s can be located at various loca- 55 
tions on the effector. If more than two speakers are 
used, such as a special room fitted with many speakers 
at many planes relative to the operator, each fsr can be 
associated with a single speaker, or a combination of 
speakers, thereby giving a direct impression of the loca- 60 
tion of the force. The amplitude of the auditory signal 
can be used to indicate the amplitude of the force. Alter- 
natively, if frequency is not used for any other indica- 
tion, varying frequency can be used to indicate force 
amplitude. 65 

Even with only a pair of speakers, such as head- 
phones, signals from various fsr’s spread out essentially 
in a plane can be mixed and sent to the pair of speakers 
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to give the user a more complicated spatial presentation. 
This would be much as a pair of headphones or speakers 
can convey to a listener the locations of musical instru- 
ments spread out over a stage in a symphonic presenta- 
tion. Frequency can be used to indicate the direction of 
forces applied other than in a single plane. For instance, 
higher frequency can indicate force being applied from 
above a reference plane, while lower frequency can 
indicate force being applied from a direction below the 
reference plane. The frequency cue may be used either 
alone or in combination with a degree of the spatial 
representation similar to symphonic representation dis- 
cussed above. 

Considering the vibrotactile modality, an example 
has been shown with reference to FIG. 7 where a map- 
ping is made between the operator’s hand and the ele- 
ments of the effector, as discussed above. With differ- 
ently shaped effectors, different numbers of sensors can 
be used, in different locations. For a different sort of 
effector, for instance a robot walking on two legs, vi- 
brotactile displays may be connected to the operator’s 
feet and/or legs, rather than the hand. Both hands or 
feet can be used. Alternatively, an array of display ele- 
ments can be worn by the user, such as in a waist belt or 
a headband, providing 360° of direction information in a 
single plane, very similar to the type of information that 
would be provided through a similar array of audio 
speakers. Use of the vibrotactile display could be made 
by an operator with hearing or seeing impairments. 

The invention may also be used to provide force 
feedback for the user of an artificial limb, such as a 
hand, gripper or foot. Taking an artificial hand as an 
example, force sensing sensors would reside in the fin- 
gertips, and other locations, such as the wrist, the force 
feedback would then be displayed, for instance, through 
vibrotactile displays on the patients skin. The display 
location could be on the stump of the limb involved, the 
opposite limb, if available, or other locations. This in- 
formation could be useful for common tasks such as 
grasping delicate objects, opening doors, or finding 
objects in the dark. 

Some of the work underlying the present invention 
was discussed in Massimino, M. J. and Sheridan, T. B., 
“Sensory Substitution of Force Feedback For the Hu- 
man-Machine Interface in Space Teleoperation,” deliv- 
ered at the World Space Congress in Washington, D.C., 
Aug. 31, 1992, available from the American Institute of 
Aeronautics and Astronautics c/o Tasco, 9 Jay Court 
P,0. Box 753 Waldorf, Md. 20604, as paper No. IAF- 
/IAA-92-0246, which is incorporated fully herein by 
reference. The invention is also described in Massimino, 
M.J., “Sensory Substitution For Force Feedback in 
Space Teleoperation,” submitted in partial fulfillment of 
the requirements for the degree of Doctor of Philoso- 
phy in Mechanical Engineering at the Massachusetts 
Institute of Technology, June 1992, which is also fully 
incorporated herein by reference. 

The foregoing discussion should be understood as 
illustrative and should not be considered to be limiting 
in any sense. While this invention has been particularly 
shown and described with references to preferred em- 
bodiments thereof, it will be understood by those skilled 
in the art that various changes in form and details may 
be made therein without departing from the spirit and 
scope of the invention as defined by the claims. 

Having described the invention, what is claimed is: 

1. An apparatus for an operator to control an effector 
that is remote from the operator to interact with an 
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environment that is also remote from the operator by 
operator manipulation of an input device in an environ- 
ment local to the operator, said apparatus comprising: 

a. at least one means for sensing an amplitude of at 
least one force arising between the remote effector 
and the remote environment; 

b. means for generating a force feedback signal that 
corresponds to the amplitude of said sensed force; 

c. means for transmitting said force feedback signal 
from said remote environment to the location of 
said operator; 

d. means for transducing said force feedback signal 
into an auditory sensory substitution signal to 
which the operator is sensitive; and 

e. means for presenting said transduced sensory sub- 
stitution signal to the auditory system of the opera- 
tor such that a full range of force amplitude may be 
represented by said means for presenting said trans- 
duced auditory sensory substitution signal. 

2. The apparatus of claim 1, said means for presenting 
said transducer sensory substitution signal comprising at 
least one acoustic sound speaker. 

3. The apparatus of claim 2, said at least one acoustic 
sound speaker comprising a pair of sound speakers as- 
sembled as a set of headphones. 

4. The apparatus of claim 1, said means for transduc- 
ing comprising means for representing variations of a 
preselected component of said force feedback signal by 
variations of the amplitude of the sensory substitution 
auditor signal. 

5. The apparatus of claim 1, said means for transduc- 
ing comprising means for representing variations of a 
preselected component of said force feedback signal by 
variations of the frequency of the sensory substitution 
auditory signal. 

6. The apparatus of claim 2, said at least one speaker 
comprising a plurality of speakers distributed around 
the operator. 

7. An apparatus for an operator to control an effector 
that is remote from the operator to interact with an 
environment that is also remote from the operator by 
operator manipulation of an input device in an environ- 
ment local to the operator, said apparatus comprising: 

a. with respect to at least one force arising between 
the remote effector and the remote environment, 
said force acting along a direction, at least one 
means for sensing said direction of said at least one 
force; 

b. means for generating a force feedback signal that 
corresponds to the direction along which said 
sensed force acts, said force feedback signal being 
characterized by at least one component having a 
frequency and an amplitude; 

c. means for transmitting said force feedback signal 
from said remote environment to the location of 
said operator; 

d. means for transducing said force feedback signal 
into an auditory sensory substitution signal to 
which the operator is sensitive; and 

e. means for presenting said transduced auditory sen- 
sory substitution signal to the auditory system of 
the operator. 

8. The apparatus of claim 7, said means for presenting 
said transduced auditory sensory substitution signal 
comprising at least one acoustic sound speaker. 

9. The apparatus of claim 8, said at least one acoustic 
sound speaker comprising a pair of sound speakers as- 
sembled as a set of headphones. 

10. The apparatus of claim 7, said means for transduc- 
ing comprising means for representing variations of a 
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preselected component of said force feedback signal by 
variations of an amplitude of the sensory substitution 
auditory signal. 

11. The apparatus of claim 10, wherein said sensory 
substitution signal is characterized by at least two com- 
ponents, each having a frequency and an amplitude, said 
means for representing variations of a preselected com- 
ponent of said force feedback signal by variations of the 
amplitude of the sensory substitution auditory signal 
comprising means for representing variations of a prese- 
lected component of said force feedback signal by a 
combination of variations of the amplitudes of selected 
of the at least two components of the sensory substitu- 
tion auditory signal. 

12. The apparatus of claim 7, said means for transduc- 
ing comprising means for representing variations of a 
preselected component of said force feedback signal by 
variations of a frequency of the sensory substitution 
auditory signal. 

13. The apparatus of claim 8, said at least one speaker 
comprising a plurality of speakers distributed around 
the operator. 

14. An apparatus for an operator to control an effec- 
tor that is remote from the operator to interact with an 
environment that is also remote from the operator by 
operator manipulation of an input device in an environ- 
ment local to the operator, said apparatus comprising: 

a. with respect to at least one force arising between 
the effector and the remote environment, said force 
acting at a location of the effector, at least one 
means for sensing said location at which said force 
acts; 

b. means for generating a feedback signal that corre- 
sponds to the location of said sensed force; 

c. means for transmitting said force feedback signal 
from said remote environment to the location of 
said operator; 

d. means for transducing said force feedback signal 
into a sensory substitution auditory signal to which 
the operator is sensitive; and 

e. means for presenting said transduced auditory sen- 
sory substitution signal to the auditory system of 
the operator. 

15. An apparatus for an operator having an auditory 
system to receive information regarding an environ- 
ment remote from the operator, said apparatus compris- 
ing: 

a. an input device located in an environment local to 
the operator; 

b. an effector located in the remote environment; 

c. means for generating a signal in response to manip- 
ulation of the input device; 

d. means for manipulating said effector in response to 
said signal representative of manipulation of the 
input device; 

e. at least one means for sensing forces arising be- 
tween the effector and the remote environment; 

f. means for generating a force feedback signal repre- 
senting a force that corresponds to said sensed 
force; 

g. means for transmitting said force feedback signal 
from said remote environment to the location of 
said operator; 

h. means for transducing said force feedback signal 
into a sensory substitution auditory signal to which 
the operator is sensitive; and 

i. means for presenting said transduced sensory substi- 
tution signal to the operator’s auditory system. 

* * * * * 
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